1* 




United States Patent and Trademark Office 



UNITED STATES DEPARTMENT OF COMMERCE 
United States Patent and Trademark Office 
Address: COMMISSIONER FOR PATENTS 
P.O. Box 1450 

Alexandria, Virginia 22313-1450 
www.uspto.gov 



APPLICATION NO. 



FILING DATE 



FIRST NAMED INVENTOR 



ATTORNEY DOCKET NO. CONFIRMATION NO. 



10/642,358 



08/15/2003 



Michael Christopher Burl 



20995 7590 02/16/2005 

KNOBBE MARTENS OLSON & BEAR LLP 
2040 MAIN STREET 
FOURTEENTH FLOOR 
IRVINE, CA 92614 



EVOL.006A 



1624 



EXAMINER 



LAROSE, COLIN M 



ART UNIT 



PAPER NUMBER 



2623 

DATE MAILED: 02/16/2005 



Please find below and/or attached an Office communication concerning this application or proceeding. 



PTO-90C (Rev. 10/03) 



Office Action Summarv 

Vfllwv ^^w*fw#f WM f fill fCtf y 


Application No. 

10/642,358 


Applicant(s) 
BURLETAL 


Examiner 

Colin M. LaRose 


Art Unit 

2623 





- The MAILING DATE of this communication appears on the cover sheet with the correspondence address - 



Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

• If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 1 33). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1 )E3 Responsive to communication(s) filed on 29 June 2004 . 
2a)Q This action is FINAL. 2b)S This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quay/e, 1935 CD. 1 1 , 453 O.G. 213. 

Disposition of Claims 

4) Kl Claim(s) 1-53 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) ^ Claim(s) 1-33 and 41-53 is/are rejected. 

7) ^ Claim(s) 34-40 is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) D The specification is objected to by the Examiner. 

10)D The drawing(s) filed on is/are: a)D accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 

Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 
1 1 )□ The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-1 52. 

Priority under 35 U.S.C. § 119 

12)D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 19(a)-(d) or (f). 
a)D All b)Q Some * c)D None of: 

1 0 Certified copies of the priority documents have been received. 

2. Q Certified copies of the priority documents have been received in Application No. . 

3. Q Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 



Attachment(s) 

1 ) ^ Notice of References Cited (PTO-892) 4) □ Interview Summary (PTO-41 3) 

2) □ Notice of Draftsperson's Patent Drawing Review (PTO-948) Paper No(s)/Mail Date. j . 

3) ^ Information Disclosure Statement(s) (PTO-1449 or PTO/SB/08) 5) □ Notice of Informal Patent Application (PTO-152) 
V Paper NofcVMail Date 0604 . 6) □ Other: . 



U.S. Patent and Trademark Office 
PTOL-326 (Rev. 1-04) 



Office Action Summary 



Part of Paper No./Mail Date 0105 



Application/Control Number: 10/642,358 Page 2 

Art Unit: 2623 

DETAILED ACTION 
Arguments and Amendments 
1 . Applicant's amendments and/or arguments filed 29 June 2004, have been entered and 
made of record. 



Response to Amendments and Arguments 
2. Applicant's remarks with respect to claims 19-26 and 50 have been considered but are 
now moot in view of the new grounds of rejection presented below. 



Claim Objections 

3. Claims 32 and 33 are objected to under 37 CFR 1 .75 as being a substantial duplicate of 
each other. When two claims in an application are duplicates or else are so close in content that 
they both cover the same thing, despite a slight difference in wording, it is proper after allowing 
one claim to object to the other as being a substantial duplicate of the allowed claim. See MPEP 
§ 706.03(k). 



Claim Rejections - 35 USC § 102 
4. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the 
basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(b) the invention was patented or described in a printed publication in this or a foreign country or in public use or on 
sale in this country, more than one year prior to the date of application for patent in the United States. 

(e) the invention was described in (1) an application for patent, published under section 122(b), by another filed 
in the United States before the invention by the applicant for patent or (2) a patent granted on an application for 
patent by another filed in the United States before the invention by the applicant for patent, except that an 
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international application filed under the treaty defined in section 351(a) shall have the effects for purposes of this 
subsection of an application filed in the United States only if the international application designated the United 
States and was published under Article 21(2) of such treaty in the English language. 

5. Claim 49 is rejected under 35 U.S.C. 102(b) as being anticipated by "A Structure-from- 
motion Algorithm for Robot Vehicle Guidance" by Wang et al. ("Wang"). 

Regarding claim 49, Wang discloses a method (figure 1) of detecting that a mobile robot 
has been kidnapped (i.e. displaced), the method comprising: 

receiving an indication that the mobile robot is not instructed to be moving (§ 5 Motion 
Detection: when the temporal differentiation is less than a threshold, it indicates that the mobile 
robot is idle and not instructed to be moving; the robot then remains in the "idle" state, as shown 
in figure 1); 

receiving data for video images from a camera coupled to the mobile robot (§ 1 
Introduction: the algorithm receives a sequence of binocular images to process); 

comparing data from different video images to determine whether or not the mobile robot 
is in motion (§ 5 Motion Detection: temporal differentiation is compared to a threshold to 
determine if the robot is in motion); and 

determining that the mobile robot has been kidnapped when the video images indicate 
that the mobile robot is in motion (§ 5 Motion Detection: when the temporal differentiation is 
greater than a threshold, it indicates that the mobile robot is moving and has been displaced, or 
"kidnapped"; the method then instantiates motion and moves to the "run" state). 



6. Claims 27, 42, 44, and 51 are rejected under 35 U.S.C. 102(e) as being anticipated by 
U.S. Patent 6,321,147 by Takeda et al. ("Takeda"). 
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Regarding claims 27 and 5 1 , Takeda discloses a method (figure 2) of controlling a 
behavior of a mobile robot based on a mismatch between an intended motional state and a 
perceived motional state in a mobile robot, the method comprising: 

receiving an indication of the intended motional state, where the motional state is selected 
from the group including moving and not moving (SI : an indication that the vehicle is intended 
to be running is received); 

using visual data from a camera that is coupled to the mobile robot to perceive the 
motional state of the mobile robot, where the perceived motional state of the mobile robot is 
selected from the group including moving and not moving (S6: a camera, coupled to the robot, 
takes image of the course in front of the robot; S7: the image is displayed and the motional state 
of the robot is perceived by an operator); 

comparing the intended motional state to the perceived motional state to detect whether a 
mismatch exists between the intended motional state and the perceived motional state (S8: an 
operator compares the image, which indicates the motional state of the robot as "not moving," to 
the intended motional state of "moving" and determines that a mismatch has occurred; that is, the 
operator perceives that, although the robot is intended to be moving, it has stopped in the face of 
an obstacle); and 

changing the behavior of the mobile robot at least partly in response to a detected 
mismatch (S9,S10: in response to the perceived stoppage of the robot, an obstacle is removed, 
and the behavior of the robot is changed so that it continues moving as intended). 

Regarding claim 42, Takeda discloses the motional state is perceived using only the 
visual data from the camera (i.e. operator uses only visual data to perceive the motional state). 
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Regarding claim 44, Takeda discloses transferring visual data from the mobile robot to a 
remote computer (column 7, lines 29-40: visual data is transmitted from the robot to the central 
monitor station 2); 

performing at least part of filtering operations in the remote computer (column 7, lines 
29-40: a user in the remote computer station views the visual data and filters the data by 
discerning movable from immovable obstacles); and 

transferring an indication of the perceived motional state of the mobile robot from the 
remote computer (column 7, lines 41-55: if the detected obstacle does not block the robot's 
passage, then an instruction indicative of the perceived and commanded motional state is 
transferred to the robot; for example, the instruction for the robot to run indicates that the 
perceived motional state is stopped). 



Claim Rejections - 35 USC § 103 

7. The text of those sections of Title 35, U.S. Code not included in this action can be found 
in a prior Office action. 

8. Claims 1, 2, and 1 1-17 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
U.S. Patent 5,838,828 by Mizuki et al. ("Mizuki") in view of U.S. Patent 5,387,947 by Shin and 
U.S. Patent 5,109,425 by Lawton. 

Regarding claim 1, Mizuki discloses a method (figures 1 and 5) of determining a 
motional state, the method comprising: 

retrieving pixel data for images taken at intervals from a camera (figure 1: video camera 
1 1 takes images of a current frame and a reference frame); 
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comparing pixel data for a first image to pixel data for a second image to generate a 
measure of a difference between the two images, wherein comparing comprises: 

filtering the first image pixel data with a gradient magnitude filter, where the 
gradient magnitude filter computes at least a spatial gradient (edge detector 18 produces a 
spatial gradient of the current frame); 

generating a binary map of the first image pixel data (column 6, lines 18-26: the 
edge detector produces a binary edge map of the current image); 

filtering the second image pixel data with the gradient magnitude filter (edge 
detector 18 produces a spatial gradient of the reference frame); 

generating a binary map of the second image pixel data (column 6, lines 18-26: 
the edge detector produces a binary edge map of the reference image); and 

comparing the binary map of the first image pixel data to the binary map of the 
second image pixel data to identify data for pixels that are different between the first 
image and the second image (binary block matcher 20 finds the pixel displacement 
between the binary edge maps of the current and reference frames, thereby identifying the 
pixels that are different between the two binary edge maps; see column 12, lines 5-9); 
using the comparison of the pixel data to count the number of pixel data identified as 
changed (column 12, lines 10-20: the displacement value between the binary edge maps denotes 
the number of mismatched, or changed, pixels) ; 

comparing the count to a third predetermined threshold (column 12, lines 20-28: the 
count of the changed pixels having the minimum distortion is compared to a threshold); and 
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determining the motional state at least partly in response to the count (column 12, lines 
20-28: if the count having the minimum distortion is lower than a predetermined threshold, than 
the motion vector is saved and is representative of the motion in the image). 

Mizuki discloses creating binary edge maps for the current and reference frames but does 
not expressly disclose comparing the gradient-magnitude filtered images to first and second 
thresholds, and then generating binary maps based on the comparisons, as claimed. 

Shin discloses a method for determining a motion vector (figures 2 and 4). Shin's method 
involves generating the spatial gradients of current and reference frames (10 and 11, figure 2). 
Then, Shin teaches that the spatial gradient maps are converted into binary maps at blocks 12 and 
13 by comparing the gradient-magnitude filtered images to thresholds and assigning a pixel-of- 
interest with the value "1" when the gradient of any pixel value in a local area is larger than the 
threshold (see column 3, lines 8-23). 

It would have been obvious to one of ordinary skill in the art at the time of the invention 
to modify Mizuki by Shin to compare the gradient-magnitude filtered images to first and second 
thresholds and then generate binary maps based on the comparisons, as claimed, since Mizuki 
utilizes binarized gradient maps, and Shin teaches that the binarization of a gradient map is 
conventionally generated by comparing the gradient map to a threshold and assigning binary 
values to the gradient map based on the comparison. 

Mizuki also does not disclose that the camera is coupled to a robot and that the method is 
for determining the motional state of a robot. 

Lawton discloses a method (see e.g. column 1, lines 15-39) whereby a video camera 
coupled to a mobile robot, such as in figure 5, captures consecutive images, and the gradient 
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maps of the images are generated and compared for the purposes of determining the motion 
present between the consecutive images. The motion between the images is then indicative of the 
motional state of the robot and is utilized to help navigate the robot. 

It would have been obvious to one of ordinary skill in the art at the time of the invention 
to modify Mizuki and Shin by Lawton to capture image from a camera couple to a robot and to 
determine the motional state of the robot, as claimed, since Lawton teaches that the motion 
between consecutive frames of video is conventionally utilized for the specific application of 
determining the motional state of a mobile robot. 

Regarding claim 2, Mizuki' s method is to be performed substantially in real time. 

Regarding claim 11, Mizuki discloses the third predetermined threshold varies at least 
partly in response to a number of pixels used to compute the count (column 12, lines 10-11). 

Regarding claim 12, Shin discloses the first threshold and the second threshold are 
adaptive to filtering by the gradient magnitude filter of the first image pixel data and the second 
image pixel data, respectively (i.e. Shin's thresholds utilized for the edge detectors (10 and 1 1) 
corresponding to the first and second image data are suitable (adaptive) for use with the gradient 
magnitude filtered data). 

Regarding claim 13, Shin does not expressly disclose the first threshold is selected to be 
about half the maximum value for the gradient magnitude computed for a pixel by the gradient 
magnitude filter after filtering of the first image pixel data, and where the second threshold is 
selected to be about half the maximum value for the gradient magnitude computed for a pixel by 
the gradient magnitude filter after filtering of the second image pixel data. However, the precise 
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value of the thresholds appears to be an arbitrary consideration and does not constitute a critical 
inventive aspect of the present invention. One of ordinary skill in the art would have expected 
Applicant's method that utilizes thresholds which are half the maximum value of the gradient 
magnitude to perform equally well with and be substantially equivalent to Shin's method 
because both are suitable for detecting binary edges in an image. 

Regarding claim 14, Mizuki discloses the comparison of the binary map of the first image 
pixel data to the binary map of the second image pixel data comprises a sloppy exclusive-or 
(XOR) computation (column 10, lines 15-25). 

Regarding claim 15, Mizuki discloses the sloppy XOR is computed by comparing a pixel 
from the binary map of the first image pixel data with a location of (x,y) to a plurality of pixels 
in the binary map of the second image pixel data, wherein the plurality of pixels includes a pixel 
with a location of (x,y) (see column 10, lines 15-42). 

Regarding claim 16, Lawton teaches that the motional state of the robot is determined 
only with observation of visual data receive from cameras 24, figure 5. 

Regarding claim 17, Lawton discloses the mobile robot is autonomous, and where the 
method is performed entirely within the mobile robot (figure 5: the method is contained 
performed in the robot). 

9. Claims 1, 3, 4, 6, and 7 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
U.S. Patent 5,838,828 by Mizuki et al. ("Mizuki") in view of U.S. Patent 5,387,947 by Shin and 
U.S. Patent 5,109,425 by Lawton, as applied to claim 1, and further in view of U.S. Patent 
6,496,592 by Lanini. 
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Regarding claims 3 and 6, Mizuki does not disclose the gradient magnitude filter further 
comprises a low-pass filter or that the image data is low-pass filtered prior to computing the 
gradient. 

Lanini discloses a method for detecting moving objects in a video sequence, which 
involves computing spatial gradients (35, figure 2). In particular, Lanini discloses that applying a 
gradient magnitude filter further comprises applying a low-pass filter 32. Lanini includes the 
low-pass filter in order to eliminate noise so that it is not subsequently emphasized by the 
gradient filter (column 2, lines 56-59). 

It would have been obvious to one of ordinary skill in the art at the time of the invention 
to modify Mizuki by Lanini to include a low-pass filter, as claimed, since Lanini discloses that 
when applying a gradient filter, it is advantageous to also apply a low-pass filter so that noise to 
reduced prior to application of the gradient filter. 

Regarding claim 4, Lanini discloses the low-pass filter corresponds to a Gaussian filter 
(column 2, lines 56-59). 

Regarding claim 7, neither Mizuki not Lanini expressly discloses that the low-pass filter 
is characterized by the following convolution kernel: [1 4 6 4 1]/16; and the gradient filter is 
characterized by the following convolution kernel: [-1 0 l]/2. However, the precise parameters of 
the filters appears to be an arbitrary consideration and does not constitute a critical inventive 
aspect of the present invention. One of ordinary skill in the art would have expected Applicant's 
method that utilizes the claimed kernels to perform equally well with and be substantially 
equivalent to Mizuki's and Lanini's filters because all are suitable for achieving the desired 
result of a smoothed and gradient-filtered image. 
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10. Claims 5 are rejected under 35 U.S.C. 103(a) as being unpatentable over U.S. Patent 
5,838,828 by Mizuki et al. ("Mizuki") in view of U.S. Patent 5,387,947 by Shin, U.S. Patent 
5,109,425 by Lawton, and U.S. Patent 6,496,592 by Lanini, as applied to claim 3, and further in 
view of U.S. Patent 5,706,355 by Raboisson et al. ("Raboisson"). 

Regarding claim 5, neither Mizuki nor Lanini discloses filtering the first image pixel data 
further comprises: filtering the first image pixel data in a first direction and in a second direction 
with a gradient filter that is characterized by the following convolution kernel, [-1 -4-5 054 
l]/32, where results of filtering are separately maintained for the first direction and for the 
second direction; and combining the separately maintained data by computing a magnitude of the 
data. 

Raboisson discloses a method for detecting obstacles in an images that involves 
analyzing sequences of road images taken by a camera on board a vehicle. In particular, 
Raboisson discloses taking separate gradients of an image in two different directions, and then 
combining the separate results to compute the gradient magnitude of the data. See figure 4. 

It would have been obvious to one of ordinary skill in the art at the time of the invention 
to modify Mizuki to compute the gradient magnitude as claimed, since Mizuki teaches that his 
edge detector may use "any of the techniques well known to those or ordinary skill in the art," 
and Raboisson discloses that gradient magnitudes are conventionally computed using the 
gradients in two separate directions, as claimed. Although Raboisson' s convolution kernel is not 
the claimed kernel, they are considered to be substantially equivalent in that both the claimed 
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kernel and Raboisson's kernel are operative to compute a directional gradient magnitude, with 
there being no unexpected results achieved from utilizing one kernel in lieu of the other. 

11. Claim 8 is rejected under 35 U.S.C. 103(a) as being unpatentable over U.S. Patent 
5,838,828 by Mizuki et al. ("Mizuki") in view of U.S. Patent 5,387,947 by Shin and U.S. Patent 
5,109,425 by Lawton, as applied to claim 1, and further in view of U.S. Patent 6,321,147 by 
Takeda et al. ("Takeda"). 

Regarding claim 8, Lawton is silent to receiving an indication that the mobile robot is 
intended to be in motion; determining that the motional state of the mobile robot is not in 
motion; and providing an indication to a user that an undesired stoppage of the mobile robot has 
occurred. 

Takeda discloses a method (figure 2) for navigating a mobile robot. In particular, Taked 
discloses receiving an indication that the mobile robot is intended to be in motion (SI); 
determining that the motional state of the mobile robot is not in motion (S3); and 
providing an indication to a user that an undesired stoppage of the mobile robot has 
occurred (S4,S6,S7). 

It would have been obvious to one of ordinary skill in the art at the time of the invention 
to modify Lawton by Takeda to indicate undesired stoppage, as claimed, since Takeda teaches 
that mobile robots may be undesirably stopped by obstacles, and therefore, there is a need to 
notify a user so that the obstacle may be removed. 
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12. Claim 9 are rejected under 35 U.S.C. 103(a) as being unpatentable over U.S. Patent 
5,838,828 by Mizuki et al. ("Mizuki") in view of U.S. Patent 5,387,947 by Shin and U.S. Patent 
5,109,425 by Lawton, as applied to claim 1, and further in view of U.S. Patent 5,001,635 by 
Yasutomi et al. ("Yasutomi"). 

Regarding claim 9, Lawton does not disclose receiving an indication that the mobile 
robot is intended to be in motion in a first direction; determining that the motional state of the 
mobile robot is not in motion while receiving the indication that the mobile robot is intended to 
be in motion; and changing a path of travel for the mobile robot at least partly in response to the 
determination. 

Yasutomi discloses a method (figure 4) for controlling a mobile robot. In particular, 
Yasutomi discloses receiving an indication that the mobile robot is intended to be in motion in a 
first direction ("start moving" indicates the robot is intended to be moving forward); 

determining that the motional state of the mobile robot is not in motion while receiving 
the indication that the mobile robot is intended to be in motion ("stop" indicates that the robot 
has stopped in the face of an obstacle even though the intended motion is to travel forward); and 

changing a path of travel for the mobile robot at least partly in response to the 
determination ("turn": the robot is instructed to change its path to avoid the obstacle). 

It would have been obvious to one of ordinary skill in the art at the time of the invention 
to modify Lawton by Yasutomi to change the path of travel of the robot, as claimed, since 
Yasumtomi teaches that that mobile robots may be undesirably stopped by obstacles, and 
therefore, there is a need to change the robot's path of travel in order to avoid the obstacles. 
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13. Claim 10 is rejected under 35 U.S.C. 103(a) as being unpatentable over U.S. Patent 
5,838,828 by Mizuki et al. ("Mizuki") in view of U.S. Patent 5,387,947 by Shin and U.S. Patent 
5,109,425 by Lawton, as applied to claim 1, and further in view of U.S. Patent 6,408,109 by 
Silver et al. ("Silver"). 

Regarding claim 10, neither Mizuki nor Lawton discloses subsampling the pixel data 
such that fewer than all the available pixels from the camera are used. 

Silver discloses a method for detecting edges in an image using image gradients. In 
particular, Silver discloses subsampling the image so that fewer than all the available pixels from 
the camera are used for determining the gradients of the image. Column 6, line 64 through 
column 7, line 4. It would have been obvious to one of ordinary skill in the art at the time of the 
invention to modify Mizuki to subsample the image, as claimed, since Silver teaches that, prior 
to determining the gradient of an image for the purposes of detecting edges, it is conventional to 
subsample the image so that the computations of the gradient are faster and less intensive. 

14. Claim 18 is rejected under 35 U.S.C. 103(a) as being unpatentable over U.S. Patent 
5,838,828 by Mizuki et al. ("Mizuki") in view of U.S. Patent 5,387,947 by Shin and U.S. Patent 
5,109,425 by Lawton, as applied to claim 1, and further in view of U.S. Patent 5,995,884 by 
Allen etal. ("Allen"). 

Regarding claim 18, Lawton does not disclose sending pixel data from the mobile robot 
to a remote computer; performing at least part of filtering operations in the remote computer; and 
receiving an indication of the motional state of the mobile robot from the remote computer. 
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Rather, it appears that Lawton discloses that all the processing of image data and 
determination of the motional state is executed via processors on-board the vehicle. 

Allen discloses a mobile robot that is controlled by an external computer. See figure 3. In 
particular, Allen teaches that data collected by the mobile robot (1) is communicated to a remote 
computer which contains a program (16) that is operative to process the received data. 
Subsequently, the control program communicates navigational instructions back to the mobile 
robot. 

It would have been obvious to one of ordinary skill in the art at the time of the invention 
to modify Lawton to achieve the claimed invention by performing the filtering operations 
remotely rather than on-board the robot, since Allen teaches that most people today already have 
personal computers, which are capable of performing the processing tasks required to navigate a 
mobile robot. Therefore, specialized hardware on board the robot is not needed and the cost and 
complexity of the robot is reduced. See column 9, lines 46 through column 10, line 45. 



15. Claims 19-25, 50, and 53 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
U.S. Patent 5,838,828 by Mizuki et al. ("Mizuki") in view of "A Structure-from-motion 
Algorithm for Robot Vehicle Guidance" by Wang et al. ("Wang"). 

Regarding claims 19 and 50, Mizuki discloses a method/circuit (figures 1 and 5) for 
determining a motional state, the method comprising: 

receiving pixel data for video images, where the video images are taken from a camera 
(video camera 11); 
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processing the pixel data for the video image to identify amounts of spatial gradient 
within a video image (edge detector 18 identifies amounts of spatial gradient in the images); 

characterizing the pixels of the image into two groups based on the amount of spatial 
gradient for each pixel (column 6, lines 18-26: the edge detector produces a binary edge map); 
and 

using the characterization of the pixels to compare a first image to the amounts of spatial 
gradients of a second video image to detect the presence of motion (binary block matcher 20 
finds the pixel displacement between the binary edge maps of the current and reference frames, 
thereby identifying the pixels that are different between the two binary edge maps and producing 
a motion vector; see column 12, lines 5-9). 

Mizuki does not disclose that the camera is mounted to a mobile robot and that the 
method is for determining the motional state of a robot including "in motion" and "not in 
motion." 

Wang discloses a method for guiding a mobile robot whereby a video camera coupled to 
a mobile robot captures consecutive images, and the temporal difference of consecutive images 
is computed for the purposes of determining the motion present between the consecutive images 
(see § 5 Motion Detection). The motion between the images is then indicative of the motional 
state of the robot - either idle or moving. 

It would have been obvious to one of ordinary skill in the art at the time of the invention 
to modify Mizuki and Wang to capture image from a camera couple to a robot and to determine 
the motional state of the robot, as claimed, since Wang teaches that the motion between 
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consecutive frames of video is conventionally utilized for the specific application of determining 
the motional state of a mobile robot. 

Regarding claim 20, Mizuki discloses arranging the pixels into a binary bit map as 
claimed. 

Regarding claim 21, Mizuki discloses the characterizing comprises comparing a binary 
value for one pixel of the first image to a group of binary values for pixels of the second image, 
wherein the group includes a pixel in the same location as the one pixel from the first image 
(column 8, lines 41-49 and figure 6: for computing distortion values for the motion 
measurements, the pixels of a present frame and compared to multiple pixels within a search 
window of a reference frame). 

Regarding claim 22, Mizuki discloses computing XOR values between the binary value 
and the groups of values (column 10, lines 34-42). 

Regarding claim 23, Mizuki discloses inspecting pixel data to evaluate whether enough 
useful spatial gradients exist for robust detection of the motional state (1 14, figure 5 A: there 
must be enough edges present in order to determine the motion information); and 

inhibiting a motional state of "not moving" at least partly in response to a determination 
that the detection of the motional state is not likely to be reliable (124, figure 5B: if it is 
determined that the motional state measurement is not reliable (i.e. there is high distortion), then 
the image portion is not assigned a motion vector and no movement is assumed). 
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Regarding claim 24, Wang discloses the motional state is determined only by analysis of 
visual data observed by the camera 24 (i.e. there are no other types of sensors or the like for 
collecting data for determining motion). 

Regarding claim 25, Wang's robot is autonomous (figure 5: planetary rover) and the 
method is performed entirely within the robot (figure 5: machine vision system is contained in 
the robot). 

Regarding claim 53, Mizuki discloses processing the pixel data to identify amounts of 
spatial gradient within the entire video image (i.e. Mizuki processes entire video images). 

16. Claim 26 is rejected under 35 U.S.C. 103(a) as being unpatentable over Lawton Mizuki 
in view of Wang, as applied to claim 19 above, and further in view of U.S. Patent Application 
Publication 2004/0017937 by Silverstein. 

Regarding claim 26, Wand does not appear to disclose sending, performing, and 
receiving, as claimed. 

Silverstein discloses a mobile robot that relies on image information for navigation 
control. In particular, Silverstein discloses that image analysis may be performed within the 
robot, or alternatively, image data may be transferred to a remote processor for analysis 
(paragraph 39). 

It would have been obvious to one skilled in the art to modify Mizuki and Wang by 
Silverstein to send image data to a remote computer, perform part of the pixel characterizing 
operations in the remote computer, and then receive an indication of movement from the remote 
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computer, since Silverstein shows that image analysis for purposes of determining the motional 
state of a robot is conventionally performed at locations remote to the robot. 



17. Claim 28 is rejected under 35 U.S.C. 103(a) as being unpatentable over U.S. Patent 
6,321,147 by Takeda et al. ("Takeda") in view of U.S. Patent 5,001,635 by Yasutomi et al. 
("Yasutomi"). 

Regarding claim 28, Takeda discloses the intended motional state is moving, further 
comprising: instructing the mobile robot to resume traveling (S10: figure 2). 

Takeda does not disclose changing the behavior comprises changing a navigated path by 
instructing the mobile robot to travel in a direction approximately opposite to that previously 
requested for at least a distance sufficient to clear an obstruction and instructing the mobile robot 
to yaw. Rather, Takeda discloses removing an obstacle (S9), and then traveling in the same path. 

Yasutomi disclose a system for controlling a mobile robot. With regards to figure 8, 
Yasutomi discloses that when the mobile robot is traveling in a forward direction (y+) and 
encounters a fixed obstacle such as a wall, the obstacle may be avoided by moving backwards 
(y-) and instructing the vehicle to yaw so that it continues moving in a different direction away 
from the obstacle (x+). 

It would have been obvious to one of ordinary skill in the art at the time of the invention 
to modify Takeda by Yasutomi to instruct the robot to travel backwards and yaw, as claimed, 
since Yasutomi discloses that these are preferred instructions for avoiding an obstacle that is 
fixed and cannot be removed by an operator (see column 4, lines 21-51). 
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18. Claims 29-33 are rejected under 35 U.S.C. 103(a) as being unpatentable over U.S. Patent 
6,321,147 by Takeda et al. ("Takeda") in view of U.S. Patent 6,809,490 by Jones et al. ("Jones"). 

Regarding claims 29-33, Takeda does not disclose that changing the behavior comprises 
shutting off motors to conserve battery life, shutting off cleaning brushes, shutting off the 
vacuum cleaner, and setting an alert. 

Jones discloses a mobile robot capable of performing cleaning functions. In particular, 
Jones discloses that when the robot is intended to be moving but becomes stuck, then escape 
behaviors such as setting an alert, shutting off motors to conserve battery life, shutting off 
cleaning brushes, and shutting off the vacuum cleaner are initiated (see column 13, lines 25-63; 
see also column 5, line 65 through column 6, line 2). 

It would have been obvious to one of ordinary skill in the art at the time of the invention 
to modify Takeda by Jones to shut up various elements of the robot when the robot becomes 
stuck or encounters an obstacle, since Jones teaches that, for a mobile cleaning robot, it is 
advantageous to shut off robot components to conserve battery life and prevent floor damage and 
set an alert for a user when trying to escape from an obstacle or entanglement. 

- 19. Claim 41 is rejected under 35 U.S.C. 103(a) as being unpatentable over U.S. Patent 
6,321,147 by Takeda et al. ("Takeda") in view of U.S. Patent 6,362,589 by Inoue et al. 
("Inoue"). 

Regarding claim 41, Takeda discloses the intended motional state is moving but does not 
discloses that changing the behavior comprises: using the indication to determine that the mobile 
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robot has been knocked over; and initiating procedures to restore the mobile robot to an upright 
position. 

Inoue discloses an autonomous robot that includes a routine (figure 9) for determining 
when the robot has been knocked over and carrying out a procedure to return the robot to an 
upright position. In particular, Inoue utilizes an indication of intended movement coupled with a 
reading from an acceleration sensor to determine that the robot has fallen. Then, once the falling 
has been determined, the robot initiates procedures to get back up. Column 13, lines 63 through 
column 14, line 22. 

It would have been obvious to one of ordinary skill in the art at the time of the invention 
to modify Takeda by Inoue to determine that the robot has been knocked over and intialize 
procedures to stand the robot upright, since Inoue teaches that when a mobile robot is knocked 
over, it is both desirable and conventional for the robot to autonomously determine that it has 
been knocked over and attempt to return to the upright state. 

20. Claim 43 is rejected under 35 U.S.C. 103(a) as being unpatentable over U.S. Patent 
6,32 1 , 1 47 by Takeda et al. ("Takeda") in view of U.S. Patent 5,08 1 ,585 by Kurami et al. 
("Kurami"). 

Regarding claim 43, Takeda discloses the mobile robot is autonomous but does not 
disclose that the method is performed entirely within the mobile robot because it requires 
sending visual data to a remote computer. 

Kurami discloses a mobile robot navigation control system (figure 1) that is performed 
entirely within the mobile robot. Among other things, the system includes obstacle avoidance 
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control 501, which utilizes image data to detect obstacles, and a CRT 603 for displaying image 
data to a user. 

It would have been obvious to one of ordinary skill in the art at the time of the invention 
to modify Takeda by Kurami so that the method is performed entirely within the mobile robot 
since Kurami discloses that it is conventional to perform all of the processing necessary to 
navigate a mobile robot entirely within the robot and not rely on remote computers. 



21. Claims 45-48 and 52 are rejected under 35 U.S.C. 103(a) as being unpatentable over U.S. 
Patent 5,001,635 by Yasutomi et al. ("Yasutomi") in view of U.S. Patent 4,628,453 by 
Kamejima et al. ("Kamejima"). 

Regarding claims 45 and 52, Yasutomi discloses a method/circuits (figures 1 A, 4, and 8) 
of controlling the motion of a self-navigating mobile robot, the method comprising: 

receiving an indication that the mobile robot is intended to be traveling in a forward 
direction (figure 4, "start moving": indicates that the robot is traveling forward in the y+ 
direction); 

determining from visual image data that the mobile robot has ceased traveling in a 
forward direction (figure 4, "obstacles forward or arrived at cleaned area?": map data is utilized 
to determine if the robot has encountered an obstacle and cannot move in a forward direction); 

discontinuing commands to propel the mobile robot in the forward direction (figure 4: 
"stop" discontinues the forward movement of the robot); 
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commanding the mobile robot to travel in a reverse direction for at least a predetermined 
distance figure 4, "determine direction": decides where the robot is to travel - figure 8 shows a 
decision process for determining the direction to move: the robot is commanded to move 
backwards for a predetermined distance at "move back one to (y-)"); 

determining that the mobile robot has traveled in the reverse direction for at least about 
the predetermined distance and discontinuing commands to propel the mobile robot in the 
reverse direction (i.e. once it is determined that the robot has moved backwards one space in the 
(y-) direction, the command to move backwards is discontinued); 

instructing the mobile robot to yaw by at least a first predetermined angle; and 
commanding the mobile robot to resume forward motion (figure 8, "next (x+)": after the robot is 
finished moving in the (y-) direction, it is commanded to yaw to the (x+) direction and continue 
moving forward in that direction). 

Yasutomi discloses that it is determined from visual image data (i.e. map data) that the 
robot has encountered an obstacle and has ceased movement. Yasutomi discloses that the map 
data is collected from ultrasonic sensors coupled to the robot (see figure 1 A). 

Yasutomi is silent to the map data being collected from a video camera coupled to the 
mobile robot. 

Kamejima discloses a mobile robot that utilizes a video camera to collect topographical 
information for the purposes of forming navigational map data for the robot. It would have been 
obvious to one of ordinary skill in the art at the time of the invention to modify Yasutomi by 
Kamejima to utilize a video camera to collect image data, as claimed, since Kamejima teaches 
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that it is advantageous to employ a video camera on a mobile robot for the purposes of collecting 
map image data, which is utilized by the robot for navigational purposes. 

Regarding claim 46, Yasutomi discloses moving backward for a predetermined distance 
but does not disclose the predetermined distance is about 0.2 meters. However, the precise 
distance to travel backwards is dependent on many factors that are taken into consideration 
during design and implementation and does not constitute a patentable advance in and of itself. 

Regarding claim 47, Yasutomi discloses the predetermined distance corresponds to at 
least an amount sufficient to permit the mobile robot to distance itself from an interfering object 
and to yaw freely around an axis without bumping into the interfering object (i.e. the robot 
moves backward a sufficient distance so that it can yaw and continue moving without bumping 
the object). 

Regarding claim 48, Yasutomi discloses the first predetermined angle is about 90 degrees 
(i.e. the robot turns from the y+ direction to the x+ direction). 



22. Claim 5 1 is rejected under 35 U.S.C. 102(e) as being anticipated by U.S. Patent 
6,321,147 by Takeda et al. ("Takeda"). 

Regarding claim 51, Takeda discloses a computer program embodied in a tangible 
medium for controlling a behavior of a mobile robot based on a mismatch between an intended 
motional state and a perceived motional state, the computer program comprising: 
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a module with instructions for receiving an indication of the intended motional state, 
where the motional state is selected from the group including moving and not moving (figure 2, 
SI: an indication that the vehicle is to be moving is received); 

a module with instructions for using visual data from a camera that is coupled to the 
mobile robot to perceive the motional state of the mobile robot, where the perceived motional 
state of the mobile robot is selected from the group including moving and not moving (figure 2, 
S6-S8: a photo from a camera coupled to the vehicle is taken and displayed so that the motional 
state of the vehicle and the possible of continuing movement in the same direction are 
perceived); 

a module with instructions for comparing the intended motional state to the perceived 
motional state to detect whether a mismatch exists between the intended motional state and the 
perceived motional state (figure 2, S8: the photo taken by the camera indicates that although the 
vehicle is intended to be moving, it has stopped because an obstacle is in the way of the vehicle; 
in response to such a mismatch, the obstacle is removed at S9); and 

a module with instructions for changing the behavior of the mobile robot at least partly in 
response to a detected mismatch (figure 2, S10: after the mismatch was detected and the obstacle 
was removed, the behavior of the vehicle is changed so that it is once again moving). 

Allowable Subject Matter 
23. Claims 34-40 are objected to as being dependent upon a rejected base claim, but would 
be allowable if rewritten in independent form including all of the limitations of the base claim 
and any intervening claims. 
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Regarding claims 34, 35, 37, and 39, Takeda does not disclose using SLAM techniques 
or that the intended motional state is not moving. Takeda also does not disclose changing the 
behavior comprises: 

disabling mapping functions for recognition of new landmarks until localization is 
achieved (claim 34); 

automatically changing the motional state to moving (claim 35); 

disabling mapping functions for recognition of new landmarks until passage of a 
predetermined amount of time (claim 37); or 

disabling mapping functions for recognition of new landmarks until a predetermined 
number of unmatched landmarks have been observed (claim 39). 

Conclusion 

24. The prior art made of record and not relied upon is considered pertinent to applicant's 
disclosure. 

"Simultaneous Localisation and Map-Building Using Active Vision" by Davison et al. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Colin M. LaRose whose telephone number is (703) 306-3489. If 
attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, Amelia 
Au, can be reached on (703) 308-6604. The fax phone number for the organization where this 
application or proceeding is assigned is (703) 872-9306. 



Application/Control Number: 10/642,358 Page 27 

Art Unit: 2623 

Any inquiry of a general nature or relating to the status of this application or proceeding 
should be directed to the TC 2600 Customer Service Office whose telephone number is (703) 
306-0377. 
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